feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption#133
Merged
jsonbailey merged 31 commits intomainfrom Apr 21, 2026
Merged
Conversation
jsonbailey
changed the title
- Each tracker now carries a runId (UUIDv4) included in all emitted events, scoping every metric to a single execution - At-most-once semantics: duplicate calls to track_duration, track_tokens, track_success/track_error, track_feedback, and track_time_to_first_token on the same tracker are dropped with a warning Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ess tracker resumption Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jsonbailey
force-pushed
the
jb/aic-2207/update-ai-sdks-billing-spec
branch
from
bdf7384 to
211ead4
Compare
…osure The run_id parameter on LDAIConfigTracker is now required (no default). UUID generation happens in the tracker_factory closure in client.py, keeping the tracker itself a plain data holder. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Break long tuple lines in client.py to stay under 120 char limit - Add required run_id parameter to LDAIConfigTracker calls in openai and langchain provider tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the redundant _tracked dict from LDAIConfigTracker. The summary already stores each metric with None as the unset sentinel, so the nil-check on summary properties serves as the at-most-once guard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New order: ld_client, run_id, config_key, variation_key, version, model_name, provider_name, context, graph_key. All call sites converted to keyword arguments for resilience against future reorders. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…oken Reorder LDAIConfigTracker.__init__ to match updated spec: context now comes before model_name and provider_name. Also fix resumption_token to omit variationKey from the JSON when it is empty, and handle the absent key when reconstructing from a token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All six at-most-once guard warnings in tracker.py now log the track data dict (runId, configKey, etc.) to aid debugging duplicate-track scenarios. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move the resumption token decoding logic from LDAIClient.create_tracker into a classmethod on LDAIConfigTracker per spec 1.1.20.2. The client method now delegates to LDAIConfigTracker.from_resumption_token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the resumption token behavior: only include variationKey in the track data dict when it has a non-empty value. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The create_tracker field on AIConfig is now always a callable that returns a working tracker, even when the config is disabled. The factory is always set to tracker_factory — callers use the enabled flag to decide whether to proceed, not the factory result. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BREAKING CHANGE: The `tracker` field has been removed from all config dataclasses (AICompletionConfig, AIJudgeConfig, AIAgentConfig). Users must now call `config.create_tracker()` to obtain a tracker instance. ManagedModel and ManagedAgent no longer accept a tracker constructor parameter — they call `create_tracker()` from the config on each invocation. The `__evaluate` return tuple no longer includes a pre-created tracker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add graphKey to the resumption token following the spec key order: runId, configKey, variationKey (if set), version, graphKey (if set). The from_resumption_token classmethod now decodes and passes graphKey to the tracker constructor. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Judge now calls self._ai_config.create_tracker() per evaluate() invocation instead of receiving a tracker at construction time. ManagedAgentGraph no longer stores or exposes a tracker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace logging.getLogger(__name__) with the SDK's shared log instance (from ldai import log) for consistency with the rest of the codebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jsonbailey
commented
Apr 17, 2026
Migrate langchain and openai provider packages from config.tracker to config.create_tracker() and fix test signatures to match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
andrewklatzke
approved these changes
Apr 17, 2026
… factory Per AIGRAPH spec 1.4.3, AgentGraphDefinition now has a create_tracker callable that returns a new AIGraphTracker per invocation instead of storing a pre-created instance. Removes get_tracker() method entirely. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_flush_final_segment and _track_tool_calls were each calling create_tracker() independently, generating new runIds that broke per-execution event correlation. Now build_node creates one tracker per node, cached in _node_trackers, and reused by all tracking methods. Adds test_same_run_id_across_token_success_and_tool_call_events to verify all node-level events for a single execution share one runId. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
run() and _build_agents() each called create_tracker() on the graph, producing two tracker instances. Now run() creates the tracker once and passes it to _build_agents() so handoff callbacks and run-level tracking share the same instance. Tests now assert graph.create_tracker is called exactly once per run and node create_tracker is called exactly once per node. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
from_resumption_token and LDAIClient.create_tracker now return ldclient.Result instead of raising ValueError on invalid tokens, letting callers handle errors without try/except. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change AgentGraphDefinition.create_tracker from Callable[[], AIGraphTracker] with default lambda: None to Optional[Callable[[], AIGraphTracker]] with default None. Guard call sites in both runners with `is not None` before invoking. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The disabled() factory on AIConfigDefault and subclasses created configs without tracker factories, breaking the spec requirement. Replace with private module-level constants in client.py, matching how js-core handles disabled configs as an internal concern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise a clear RuntimeError if create_tracker returns None rather than letting it crash with AttributeError on track_metrics_of_async. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Cache node trackers in langgraph_callback_handler flush() to avoid creating multiple trackers per node with different runIds - Read graph key directly from config instead of instantiating a tracker just for debug logging in langgraph_agent_graph_runner - Simplify redundant except (json.JSONDecodeError, Exception) to except Exception in tracker.py from_resumption_token Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
AIConfig.create_tracker is now a required field with no default value. The SDK client always injects a real tracker factory, so any direct construction of AIConfig subclasses must now provide one explicitly. This eliminates the entire class of null-safety issues around tracker factories. Reverts the RuntimeError guard in Judge.evaluate() since it is no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convenience factory for the common fallback case. Added to AIConfigDefault, AICompletionConfigDefault, AIAgentConfigDefault, and AIJudgeConfigDefault. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace private _DISABLED_*_DEFAULT constants and inline AIXxxConfigDefault(enabled=False) calls with the new disabled() classmethod. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t overrides The base AIConfigDefault.disabled() already uses cls(), so subclass overrides were unnecessary. Use Self return type annotation for correct narrowing and remove the three identical overrides from AICompletionConfigDefault, AIAgentConfigDefault, AIJudgeConfigDefault. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
mypy targets python_version = "3.10" but typing.Self was added in 3.11. Use unconditional typing_extensions import and remove the unnecessary from __future__ import annotations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change `if self._graph_key is not None` to `if self._graph_key` so that empty string is treated as "not set", matching the truthy check already used in resumption_token. Prevents round-trip data loss where a tracker with graph_key="" emits graphKey in events but omits it from the token. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 35aef5e. Configure here.
| config_tracker = node.get_config().create_tracker() | ||
| if not config_tracker: | ||
| continue | ||
| node_trackers[node_key] = config_tracker |
There was a problem hiding this comment.
Flush always tracks success even for failed nodes
Medium Severity
The flush() method unconditionally calls config_tracker.track_success() for every node in the execution path. Combined with the new at-most-once guard on track_success/track_error, this means that if a node actually failed, track_success() is still called first and locks out any subsequent track_error() call. With the old code (no at-most-once guards), external code could potentially override this, but now the guard permanently records success for every flushed node regardless of actual outcome.
Reviewed by Cursor Bugbot for commit 35aef5e. Configure here.
Contributor
Author
There was a problem hiding this comment.
We need to rework some of this logic and will do it in a separate PR. This is currently experimental.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
runId: Every tracker now includes a uniquerunId(UUID) in all track event payloads, enabling billing isolation per executioncreate_tracker()factory on config objects:AICompletionConfig,AIAgentConfig, andAIJudgeConfignow carry an optionalcreate_trackercallable that returns a freshLDAIConfigTrackerwith a newrunIdeach time it's called. Set toNonewhen the config is disabled.ManagedModel.invoke(),ManagedAgent.run(), andJudge.evaluate()now callcreate_tracker()at the start of each invocation to get a fresh tracker, fixing the multi-turn tracking issue where at-most-once guards blocked metrics from second+ invocationsresumption_tokenproperty on tracker: URL-safe Base64-encoded (no padding) JSON string containing{runId, configKey, variationKey, version}for cross-process tracker reconstructionLDAIClient.create_tracker(token, context): Reconstructs a tracker from a resumption token for deferred feedback scenarios. Validates required fields and raisesValueErrorfor invalid tokens.Test plan
create_trackercallable; disabled config hasNonecreate_tracker()call returns a new tracker with a distinctrunIdManagedAgent.run()usescreate_tracker()when available, falls back to stored trackercreate_tracker(token, context)reconstructs tracker with originalrunIdand empty model/providerValueError🤖 Generated with Claude Code
Note
Medium Risk
Touches core SDK tracking primitives and all runner integrations, so regressions could affect metrics emission and correlation (especially around tracker factory lifetimes and duplicate-suppression behavior).
Overview
This PR reworks tracking to be per execution by replacing stored
trackerinstances on configs/graphs withcreate_tracker()factories, and updating managed wrappers (ManagedModel,ManagedAgent,Judge,ManagedAgentGraph) and both LangChain/LangGraph and OpenAI agent-graph runners to create (and in OpenAI, cache) trackers per run/node.It also adds a
runIdto everyLDAIConfigTrackerevent, enforces at-most-once semantics for key metric types (dropping duplicates with warnings), and introduces tracker resumption viaLDAIConfigTracker.resumption_token+LDAIClient.create_tracker(token, context)for deferred/cross-process feedback. Tests are updated broadly to use factories, validate runId consistency/uniqueness, ensure graph/node tracking uses the correct single tracker per run, and cover resumption-token encoding/decoding and error handling.Reviewed by Cursor Bugbot for commit 35aef5e. Bugbot is set up for automated code reviews on this repo. Configure here.